Stereotactic optical navigation

ABSTRACT

A system for quantitative computer graphic determination of positions on a patient&#39;s anatomy and positions on associated equipment located near the patient&#39;s anatomy in relation to anatomical data, as from CT or MR scanning. A first camera produces a quantitative electronic readout of its field-of-view which provides a determination of relative spacial coordinates of uniquely identifiable points in its field-of-view. A second camera produces a quantitative electronic readout of its field-of-view which provides a determination of relative spacial coordinates of uniquely identifiable points in its field-of-view. The two cameras are located with respect to the patient&#39;s anatomy and the associated equipment so that the fields-of-view of the cameras include both the patient&#39;s anatomy and the equipment, but are taken from different directions. A body marker is positioned with respect to the patient&#39;s anatomy at a known position relative to said patient anatomy. The body marker has known coordinates in a stereotactic coordinate system that is established relative to the patient&#39;s anatomy so that the coordinates of all identifiable points in the fields of view of the two cameras can be determined relative to the stereotactic coordinate system and related to imaging data.

This is a continuation of application Ser. No. 08/441,788, filed May 16,1995, issued as U.S. Pat. No. 5,662,111, which is a continuation ofapplication Ser. No. 08/299,987, filed Sep. 1, 1994, now abandoned whichis a continuation of application Ser. No. 08/047,879, filed Apr. 15,1993, now abandoned, which is a continuation of application Ser. No.07/941,863, filed Sep. 8, 1992, now abandoned, which is a continuationof application Ser. No. 07/647,463, filed Jan. 28, 1991, now abandoned.

BACKGROUND TO THE INVENTION

The concept of frameless stereotaxy is now emerging in the field ofneurosurgery. What is meant by this is quantitative determination ofanatomical positions on, let us say, the head based on data taken from aCT, MRI or other scanning needs. The data from the image scan can be putinto a computer and the head represented, according to this graphicinformation. It is useful for the surgeon to know where he will beoperating relative to this data field. He can, thus, plan his operationquantitatively based on the anatomy as visualized form the image data.Until now the use of stereotactic head frames for fixation means iscommonplace. For example, see U.S. Pat. No. 4,608,977, issued Sep. 2,1986, and entitled: System Using Computed Tomography As For SelectiveBody Treatment, Brown. These employ a head fixation device typicallywith an index means that can be visualized in scan slices or image data.Thus, the anatomical stereotactic data so determined can be quantifiedrelative to the head frame. Arc systems or probe carriers are typicallyused to direct a probe quantitatively based on this data relative to thehead holder and, thus, to the anatomy. If the surgeon can be freed fromthe use of the head holder and localizer, and still relate positions inthe anatomy to things seen on the scan or image data, then this canspare patient discomfort and could be potentially used for generalneurosurgery where only approximate target positioning is needed. Forexample, a space pointer which could be directed to the anatomy and itsposition could be quantified relative to the stereotactic image data.This space pointer, analogous to a pencil, might be therefore pointed ata position on the anatomy and the position and the direction of thepointer, subsequently appear, on the computer graphics display of theanatomical data. Such apparatus has been proposed, using an articulatedspace pointer with a mechanical linkage. In that regard, see an articleentitled “An Articulated Neurosurgical Navigation System Using MRI andCT Images,” IEEE Transactions on Biomedical Engineering, Volume 35, No.2, February 1988 (Kosugi et al), incorporated by reference herein. Itwould be convenient if this space pointer were mechanically decoupled orminimally mechanically coupled. Until now, several attempts have beenmade to implement a passive or active robotic pointer as described inthe referenced article, essentially, which consists of a pencil attachedto an articulating arm, the arm having encoded joints which providedigital angular data. Such a robotic space pointer is a mechanicallyattached device and once calibrated can give the graphic representationof the pointer on a computer screen relative to the stereotactic data ofthe head.

One objective of the present invention is to provide a camera apparatus(optical) which can visualize a surgical field and digitize the viewinformation from the camera and relate it via computer graphics means toimage data which has been taken of the patient's anatomy by imagescanning means (tomographic scanner). The relationship of the opticalcamera view and the image data will then make quantitative the anatomyseen in the camera view and also make quantitative the position ofsurgical instruments such as probes, microscopes, or space pointers tothe anatomy via the registration of the camera view to the image data.

Another objective of the present invention is to make an opticallycoupled space pointer which accomplishes the same objectives as therobotic arm mechanically coupled space pointer, e.g., give ongoingpositional correspondence between a position in a patient's brain andthe tomographic image (see Kosugi et al). The optical coupling wouldfree the surgeon from any sterility questions, provide anobstruction-free device, and avoid the encumbrances of a bulkymechanically coupled instrument.

DESCRIPTION OF THE FIGURES

FIG. 1 shows one embodiment of the present invention which involves twovideo cameras and a space pointer with two light sources on it.

FIG. 2 shows an embodiment of the present invention with the spacepointer pointing into a cranial operative site and where more than twovideo cameras are looking at the space pointer for redundancy.

FIG. 3 shows an embodiment of the present invention in which lightsources at a distance from the pointer are used to reflect offreflectors on the pointer and the reflected light is detected by videocameras to ascertain the orientation of the space pointer.

FIG. 4 shows an embodiment of the present invention in which two camerasare used and they visualize the anatomical field together with a spacepointer, index marks on the patient's anatomy and a microscope so as torelate the position and aim of the microscope to the anatomy and thespace pointer.

FIG. 5 shows a generalized, signal camera embodiment of the inventionwhere the camera is coupled to a computer graphic means and the view ofthe camera looking at the patient anatomy is related to image data fromimage scan means so as to register the camera view and the image data toquantify the camera view field.

FIG. 6 shows a schematic representation of how camera field data wouldbe registered in position and orientation to analogous image scan dataon the same computer graphic display.

FIG. 7 shows a schematic view of two cameras looking at the anatomicalsubject with corresponding graphic views both of the camera, readout andfield of view and of the computer graphic representation of the sameview.

DESCRIPTION OF THE INVENTION

FIG. 1 shows a schematic view of one embodiment of the invention. Thesetting is neurosurgery and the patient's head 6 is being operated onthrough a skull hole 7. Probe 1 is being put to the patient's head andit is desired to know the relationship of that probe 1 to the anatomy ofthe patient's head 6 as visualized from some imaging means such as CT orMR scanners or angiographic X-rays views. This image data representationof the patient's head would previously have been accumulated in acomputer, see the referenced U.S. Pat. No. 4,608,977.

The cameras 4 and 5 may, for example take the form of known devices,e.g., CCD Type compact TV Scanners with high resolution that can beeasily digitized and video displayed or displayed on computer graphicscreens, see FIG. 5. The cameras 4 and 5 may operate as disclosed in abook: Digital Image Processing, Second Edition, Addison-WesleyPublishing Company, Gonzalez and Wintz, 1987, incorporated by referenceherein. Specifically, using stereoscopic imaging techniques to mappoints sensed in a world coordinate system is treated in a section 2.5.5entitled “Stereo Imaging,” beginning on page 52 . As explained in thebook, the two cameras 4 and 5 are used to find the coordinates (X, Y andZ) of light sources 2 and 3. The subject also is treated in a book:Visualization of Natural Phenomena, by Robert S. Wolff and Larry Yaeger,First Edition, TELOS, The Electronic Library of Science, Santa Clara,Calif., 1993 (which is an imprint of Springer Verlag, New York),incorporated by reference herein. Specifically, see Chapter 3 entitled“Through Canyons and Planets,” pages 66 and 67. Detailed treatment ofcameras as imaging trackers appears in a book: The Infrared Handbook,incorporated by reference herein and prepared by the EnvironmentalResearch Institute of Michigan (1978) for the Office of Naval Research,see pages 22-63 through 22-77. See also, a book: Digital ImageProcessing, Prentice-Hall, Inc., by Kenneth R. Castleman, published inEnglewood Cliffs, N.J., 1979, incorporated by reference herein andspecifically a section entitled “Stereometric Ranging,” beginning onpage 364.

In FIG. 1, the cameras 4 and 5 are looking at the field including thepatient's head 6 and the probe 1.

The orientation and quantification of the camera coordinate data takenfrom the scan images in the video cameras can be registered by indexspots 8A, 8B and 8C placed on the patient's head. An alternative tothese index spots might be a head ring which is fixed firmly to thepatient's skull as is commonly done in surgery and that headring mayhave index points or lines on it which can be seen in the two views fromthe cameras 4 and 5. When the index points are in view of the cameras 4and 5, the appropriate transformations can be made if the coordinates ofthe physical points 8A, 8B, and 8C are known beforehand to the entiredata set (CT or MR) of anatomy in the computer as indicated. Thus, thereference points are used to relate the camera data to the storedanatomical data coordinates. More than three points may also be used forredundancy or better field of view. As indicated, the probe in FIG. 1has two index light sources 2 and 3, which are also visible within acertain range to the cameras 4 and 5. Thus, the orientation of the lightsources 2 and 3 relative to the anatomy is registered by the two cameras4 and 5 and thus physical orientation of probe 1 relative to the storedCT or MR data on the head 6 is known. Since light sources 2 and 3 may bein a predetermined orientation relative to the tip 9 of the probe 1, theactual physical location of the tip 9 relative to the anatomy may alsobe computed by the data of the two views of the cameras 4 and 5. As theprobe moves in the field in front of the two cameras which are pointingin independent directions towards the probe, the position of the probecan be tracked by the video or digitized position of the two lightsources. Thus, a plurality of cameras can be used to provide real timeoptical sensing.

With the locations of the sources 2 and 3 specified, the orientation ofthe probe 1 may also be determined from these two camera views. Thus, itis possible to display by the data accumulated by the cameras 4 and 5,the orientation and absolute position of the probe 1 relative to theanatomy data, and this display can be made in computer graphics realtime as the probe 1 is moved around in a field near the anatomy e.g.,the head 6. In particular, the probe 1 position within the entry hole 7is known, and thus the tip 9 can be graphically visualized on a computerdisplay relative to the stored anatomy inside the patient's head. Thisis most useful when exploring the interior of a surgical hole when thesurgeon wishes to know the advancement of his probe or surgicalinstruments within that hole. Such an instrument may also be useful inplanning the position of a surgical incision. By pointing the probe atthe patient's skin and being able to visualize the position on the skinrelative to relevant anatomy inside the head, the surgeon can make ajudicious choice of entry point.

The light sources 2 and 3 may be LED light sources of very smalldimension and they can be powered by an internal battery in the probe 1.The probe may thus be mechanically decoupled from other apparatus andonly optically coupled through the cameras 4 and 5. This opticalcoupling can be done in other ways. For example, there may be externallight sources positioned nearby which can be reflected by tinyreflectors that function as the light sources 2 and 3 on the probe. Thereflected light can then be detected by cameras 4 and 5 giving the sameoptical registration of the probe position as though the light sources 2and 3 were sources of direct light from the probe itself.

Recalibration of the entire optical system is also possible. Cameras 4and 5 may have principle optical axes, 25 and 26 respectively shown inFIG. 1. The cameras can be aligned to point in a plane and directedtowards a common isocenter 29. Thus all rays in the field such as rays21 and 22 as seen from camera 4 to points 2 and 3 or rays 23 and 24which also connect points 2 and 3 on the probe to the camera 5 can becalibrated in the field of the cameras so that their exact anglesrelative to the principle rays indicated by 25 and 26 can bequantitatively determined. Once the quantitative orientation of theserays to the fiducial points 2 and 3 are digitized and determinednumerically, then the position and orientation of the probe 1 can becalculated relative to the point 29 which as been recalibrated asexplained below. The exact referencing of the coordinate systemrepresented by axes 25 and 26 with their crossover point 29 andorthogonal axis 27 can be determined by further fiducial points on theanatomy itself. Natural anatomical fiducial points can be used such asthe tip of the nose, the ears or other bony landmarks. However, specificindex points such as 8A, 8B, and 8C can be placed on the patient'sscalp, for example, and these used as a reference transformation set torelate the data seen by the cameras to anatomical data determined fromthe imaging. For example, the exact coordinates of the points 8A, 8B,and 8C may have been determined in space from the scan data previously.By knowing their exact coordinates in space and knowing the position ofother anatomy relative to them, by determining the position as seen bythe cameras 4 and 5 of these three fiducial points, the rest of theanatomy can also be registered in the cameras field. Thus the exactpositioning of these fiducial points onto the graphic display of theanatomical data from the images can be made. Furthermore, the exactpositioning of the probe with its fiducial points 2 and 3 can be thusset quantitatively into the field in a similar way. This operationcorresponds to a series of 3-dimensional coordinate transformations andis a straight-forward mathematical matter. Specifically, mathematicaltransformations are well known in the computer graphics prior art astreated in the textbook: Fundamentals of Interactive Computer Graphics,Addison-Wesley Publishing Company, 1982, Foley and Van Dam, incorporatedby reference herein, see Chapter 7 entitled “GeometricalTransformations.”

FIG. 2 illustrates another embodiment to the present invention in whichmore than two cameras are involved. Cameras 204 and 205, as well ascamera 210, are present and may prealigned or not prealigned prior tosurgery. They are anchored on a support structure 230 which holds themrigidly in place and that support, in turn, is clamped by means ofclamping means 231 to some stable object relative to the patient's head206 such as the operating room table or the floor itself. Headholder 232may be a standard headholder as used in most operations with pinfixation points to the skull illustrated by 233, it too can be anchoredto the operating table or to the floor by post 234 and, thus, theoptical system above it and the head holder are stabilized relative toeach other by means of their attachment to either themselves or to theoperating table. Again, the index points 202 and 203 (light sources)represent the fiducial points for the cameras 204, 205 and 210 and bydigitizing the field of these cameras, one can determine the positionand orientation of the probe 201 in space coordinates. In addition,there are the index reference points 208A, B, and C which representindependent fiducial points on the patient's head which can be alsoobserved by the cameras and the cameras can, therefore, check thestability as well as their coordinate reference frame continuously bymonitoring these fiducial points on the anatomy itself. There is atypical range of motion of the probe 201 which is practical in suchoperations and this is illustrated as an example by the dashed-line cone240. It must be that the cameras can visualize the probe 201 and thefiducial points 202 and 203 everywhere within the working cone 240. Thisis typically the range in which the surgeon will be introducinginstruments into the cranial opening site 207. It is clear that thepositions of the cameras 204, 205 and 210 can be prearranged andprecalibrated on the bar 230. This may be done so that they are pointingisocentrically to the same point in that their visualization fields areprecalibrated and preoriented so that everything within the field has aknown calibration. This could also be easily checked by taking theplatform 230 off at any given time and putting it on a phantom base orsome other jig structure which enables instant calibration of thesystem. It is also true that the head holder 232 may have fiduciallights on it or fiducial points 233A, 233B and 233C so that it may bereferenced relative to the cameras and the entire system becomes anintegral digitized calibrated system.

FIG. 3 shows another embodiment of the present invention in whichexternal light sources 341 and 342 are present as well as the cameras304 and 305 for receiving optical signals. Cameras 304 and 305 arearranged and fixed to a bar 330 for positioning. Light sources 342 and341 are also arranged and attached to the bar 330 so that they aimtowards the probe 301 which has reflectors on it, specifically sources302 and 303 which reflect the light from the light sources 341 and 342.The cameras 304 and 305 detect the reflected light which is illustratedby the dashed-line light beams shown in FIG. 3. In this way the probe301 does not have to have any energy source or active light sources, butcan be merely a reflector of light. It is also true that the probeitself could be one, long reflective linear arrangement or could haveother arrangements of the fiducial points instead of the lineararrangement 302, which is coaxial with the probe. Any kind of patternrecognition of this type could be detected by the cameras 304 and 305and the corresponding digitization of the probe position and orientationcould be made.

In this example of FIG. 3 we also show headring 350 which is affixed tothe patient's head by a series of head posts 356 anchored securely tothe skull. On the headring are fiducial elements 351, 352 and 353 whichserve as index points and reference points that can also be detectedoptically by the cameras 304 and 305. In this way, the ring 350represents a platform and corresponding coordinate system basis, theposition of the coordinate system being referenced by the fiducialpoints 351, 352 and 353 and monitored in terms of its relative positionto the bar 330 and its associated cameras. In this way the entireoperative setting could be monitored for any differences in position andposition differences can be corrected for if they are determined by thecomputer graphics associated with the cameras 304 and 305. It is notablethat the need for discrete index points such as 302 and 303 on the spacepointer is not absolutely necessary. Pattern recognition algorithms in acomputer from data from cameras 304 and 305 may simply recognize theshape of the space pointer 301. Thus, the quantitation of its positionin the field need not be done by discrete index points on theinstrument.

The major advantage of the probe structures illustrated in FIGS. 1, 2and 3 is that they are mechanically decoupled from the observing camerasand thus there are no encumbrances of mechanical linkages such as arobotic arm as has been proposed in the past. It is also true that theseprobes can be made relatively simply and to be disposable so that thesurgeon can throw the probe away after the procedure without incurringgreat expense.

FIG. 4 shows another embodiment of the present invention for use withoptical digitizing viewing means which involves not only a probe 401,but also an operating microscope 460. The objective here is to determinequantitatively the relationship between the patient's head 406 and itsanatomy within it, the space probe 401 and the operating microscope 460.The principle is essentially the same. The patient's head 406 isstabilized by the headholder 432. The microscope has index means 462 and463 which may be LED point light sources as explained above. Similarly,the probe 401 has its index points 402 and 403. Cameras 404 and 405 areaffixed to base platform 430 and view the entire field, microscope plusprobe plus patient's head. Optical index points 408A, 408B, and 408C maybe attached to the patient's scalp or to the headholder (points 408A′,408B′ and 408C′) to provide referencing to the anatomy of both the probeand the microscope. By this sort of viewing, the relationship of theposition of the microscope 460 and its orientation relative to theanatomy can be determined as explained above. Thus, one can display on agraphics means the field of view in which the microscope is viewingrelative to the anatomy. In that regard, see the above referencedtextbook, Computer Graphics: Principles and Practice. Accordingly, whencomputer graphics representations of the anatomy have been made, thencomputer graphics of the field view with a microscope can also berepresented on the graphics display means and, thus, the relationshipbetween what the surgeon 461 is seeing and the computer reconstructedfield may be made. This is very important in planning as well asinteractive surgical resections. At the same time, the probe 401 may beinserted into the field and the position of its tip 409 can berepresented within the actual microscopic viewing field of themicroscope 460. The entire surgical array of instruments may berepresented graphically so that interactive correction and management ofthe operation can be made by the computer systems. One can also putother instruments within the field such as scalpels, probes and otherdevices which the surgeon commonly uses, these being indexed by fiducialmarks or simply visualized directly by the cameras and representationsof them put onto the graphics display means.

Thus by the index points that we have alluded to in FIGS. 1 through 4and the associated embodiments, one can relate the various structuresincluding anatomy, probes, microscopes and other instruments together inone graphics display. It should also be said that once this relationshiphas been established, then the cameras which see the actual objectsthemselves can make direct overlays of the objects as seen with thegraphic representation of these objects as calculated from the imagingprior to surgery. Thus, direct correspondence of shapes and objects canbe instantly ascertained by the operator by merely overlaying thegraphics display and the actual display together on the same graphicsscreen.

There are many variations of the embodiments shown in FIGS. 1 through 4.One does not need to have, for example, two video cameras or two or morevideo cameras pointing in the same plane. They could be non-coplanar andthere could be an array of them to encompass a much larger field ofspace. Such a multi-camera display could be precalibrated or notprecalibrated. The cameras could be monitored and stabilized by fixedfiducial points somewhere in the field so that the entire registrationand synchronization of all cameras would be possible. The mounting onwhich the cameras are held could be movable and changed interoperativelyto optimize the position of the cameras while maintaining registrationwith the subject field. The orientation of the cameras relative to theanatomy, microscope or probe could also be done without the need forfiducial lights such as sources 2 and 3 in FIG. 1 or index fiducialpoints 8A, 8B, and 8C in FIG. 1. Overall correspondence of the shape ofthe subject as viewed by the camera could be overlaid and optimized inits matching to the graphics representation of the anatomy taken fromthe images. Graphic rotation of the image data could be done so as toregister the direction of view of the camera relative to the anatomy.This correspondence would then be done by shapes of subjects in the realfield vs. shapes of subjects in the graphics field. Such optimization ofthe two shapes could be done and the direction of the camera therebydetermined relative to the field of view. Once that is done, theorientation of the probe 1 or any other shaped object related to a probecould similarly be registered from the camera's point of view. Patternrecognition algorithms be used to determine the orientation of the probe1 therefore relative to the orientation of the other subjects such asthe head and its orientation relative to the cameras.

The present invention also recognizes the use of one optical camera.Although the examples above illustrate use of two or more cameras, thereis utility in even using just one camera to view the surgical field. Itcan give you a two-dimensional representation in a projected view of thefield. One can use this representation and the graphic representationfrom the image data to register the two views and, thus, align thegraphic display in a “camera view.” Thus pointers in the field of thecamera can be registered directly on to the graphic display view. Forexample, a pointer moving on the surface of the skin would be registeredrelative to the graphic view so that you would know where that point ismoving relative to this quantitative data that represents the skin andother anatomical structures below the skin. This would have more limitedusefulness, but it could also be important. Thus, the application ofmounting a single video camera to view a surgical field and representingthat visual field on a graphic field so as to bring the two fields intoalignment by manipulation of the graphic field in the computer hasutility in the surgical setting.

FIG. 5 illustrates more specifically the use of one optical viewingcamera and registration of its field by computer graphics to image data.In FIG. 5, a camera 505 which has been anchored via arm 550 near thesurgical field, views the patient's head 506 and other objects nearby.The camera 505 is connected via cable 551 to a computer graphics displayunit incorporating a screen 552. The computer graphics screen 552 iscooperatively connected to computer calculation means and storage meansrepresented by a box 554 to produce an image as represented on thescreen. The data in the storage means (in box 554) may be provided froma scanning source, e.g., a CT or MRI scanner or it may be a magnetictape with corresponding data on it. The camera 505 is viewing the headand a representation of the head shows on the screen 552 together withimage data indicated by the contours 553. In the field, it is probe 501which is seen as representation 555 on the screen. Also, there is asurgical opening 507 and for completeness, the index marks 508A, 508B,and 508C which may aid in orienting what is seen by camera 505 to thegraphics image data seen on screen 552. The headholder 532 andassociated pins 533 hold firmly the head 506 relative to the camera 505.As shown on the screen 552, the corresponding index points 558A, 558B,and 558C are shown on the screen as well as the actual image of theanatomy and the space probe represented by image 553. Thus, if computergraphics representations of the same anatomy are simultaneously put onthe screen, for example, in a different color, then those image data canbe scaled, translated, and rotated such that they register with what isseen by the field of view of camera 505. By so doing, one has inperspective view a registration of the camera data with the image data.Thus when one looks at the probe representation 555, on the computergraphic screen 552 of the actual probe 501, one can see immediately thecorrespondence of that probe relative to the quantitative stereotacticimage data anatomy. Thus in perspective view, one is relating theposition of the probe to that stereotactic image data anatomy, and thiscan be a very useful adjunct to surgery. For example, if one wished toknow where to make the surgical opening 507, one could move the probe501 in actual space relative to the anatomy until one sees the probe inperspective view with its tip over the desired point relative to theimage data anatomy seen on screen 552. That would instantly tell youthat this is the place to make the surgical bone opening, for example.There are many other illustrations of the use and power of thisone-camera approach.

FIG. 6 shows how one might register camera anatomical data to imagemachine-acquired anatomical data as described in the paragraph relatedto FIG. 5. For example, in FIG. 6 the outline 606 represents the actualcontour of the patient's head as seen by the camera 505 in FIG. 5. Also,the points 608A and 608B and 608C are shown as dots and these too areseen by the camera. Furthermore, anatomical landmarks such as 672, thetip of the ear, and 670, the tip of the nose, may be seen by the camera505 in FIG. 5. The dashed-line contour in FIG. 6 shows a similar contourreconstructed in a perspective view from, for example, CT slice imagedata. Such image data can be stacked, can be surface rendered, and canbe viewed and oriented from any different direction by computer graphicsmanipulation. Thus, it is possible to take such “dashed” image datarepresentations, scale them proportionately, rotate them in space, andtranslate them, such that when you view the dashed and undashed contourson the computer graphics console, the operator can easily trim in theimage data or the dashed line 666 such that it matches exactly the solidline 606 on the computer graphics screen. Such treatments of computergraphics data are disclosed in a textbook: Principles of InteractiveComputer Graphics, McGraw-Hill Book Company, Newman and Sproul, 1979,incorporated by reference herein. For example, moving parts of an imageis specifically treated in a section 17.3 at page 254. Also, in asimilar way, one can make computer graphic manipulations to register thecorrespondence of the image points from the camera 608A, 608B, and 608Cwith the corresponding index points 668A, 668B, and 668C, which areillustrated by dashed points in FIG. 6. Registering these two sets ofthe same physical points in the computer graphics would be anattractable way of registering the entire two perspective views.Similarly, anatomical landmarks which are identifiable such as thecomputer graphic representation of the tip of the ear 673 and the tip ofthe nose 671 can be represented and corresponded to the analogous points672 and 670 from the camera data. The use of different colors, colorwashes, color transparencies, and other powerful graphic standards aswell as mathematical algorithms to optimize the correspondence of thesetwo perspective views are easily put into play at this point to do thejob.

FIG. 7 illustrates another embodiment of how more than one camera can beused for computer graphic registration and corresponding quantificationof an optical view. In the upper portion, one sees two cameras 704 and705, pointing at arbitrary non-identical directions towards the subject706. The fields of view are shown with the dashed lines. There is acranial hole 707 with a probe 701 in it to the depth of the brain withthe tip 709 inside the head. Index points 702 and 703 on the probe mayor may not be present and are analogous to those discussed in FIG. 1.Each of the cameras will have views as illustrated in the lower portionof FIG. 7 and are displayed on the computer graphic display means 760and 770. The display means 760 represents, for example, the view ofcamera 704 and one sees the solid line 766 which is the optical outlineas seen by camera 704 of the patient's head. Similarly, the probe 761 isseen through the burr hole 767. By computer graphic translation,rotation and scaling, one can adjust the computer graphic view so thatit matches the anatomical view, i.e. the computer graphic perimeter 766Aindicated as dash line exactly matches 766. In this way, one knows thatone has reproduced graphically with the dashed curve the projected viewas seen by 704. Analogously, camera 705 will have its view as seen ingraphic display 770 of the outline of the head 776 being matched to thegraphic outline of the head 776A. Obviously, index marks, grids or lineson the patient's scalp might help in this registration of the two cameraviews. Once these views, however, have been registered, uniquelyidentifiable points in both views can give information on the exact3-dimensional coordinates of those identifiable points relative to theanatomy as seen from the image data. For example, the points 763 and 773are identical and correspond to the physical point 702 on the probe. Oneach of the views 760 and 770 this point represents a projected line asseen from the respective camera. The two lines from the two camerasintersect at a unique point and this can easily be determined as aunique 3-dimensional point referenced to the data from the image scanneras stored in the computer. Thus, the two points 702 and 703 can bedetermined quantitatively in space relative to the anatomical data, andthus, the quantitative position of the probe and any point on the probecan be determined relative to the image data. In particular, the end ofthe probe represented by point 709 which is in the depth of the brainand indicated on the graphics display as 769 and 779 respectively can bedetermined, i.e. the 3-dimensional coordinates of that point relative tothe 3-dimensional image anatomy can be determined. Thus, there is noparticular need for special index marks as shown in FIG. 1. Mereregistration of existing anatomical structures relative to camera viewand the image data would be sufficient for a full 3-dimensionalrepresentation of any instrument such as the probe in FIG. 7 relative tothe anatomy. Using special angles such as 90° or stereoscopic views ofthe cameras could be convenient for such 3-dimensional registrationwithout prior calibration.

It also should be said that for fixed camera positions, the subjectitself might be moved so that his optical representation matches thegraphic representation. In most cases, it would seem simpler to do themovement of the subject's image data via software, than moving theanatomical subject relative to the cameras, however, both methods couldbe used for registration of the respective images.

The use of such camera registration with image data eliminates any needof camera field calibration or the need to know relative camera angles.

It should be stated that this technique and the rest of the discussionabove is differentiated from a previous attempt at registration ofcomputer graphics to anatomical viewing. This was done by Patrick Kellyin the 1980's, and is reported in the literature in several places.Kelly's approach was to move a surgical microscope to a direction thatwas determined by image scan data. He would then take reconstructedstructures from the image data and project it on a “heads up” display sothat the surgeon looking on the microscope could see a graphicrepresentation of what he should be viewing in the microscope field. Theprocedure of Kelly's was to first calculate the position from graphicsof his microscope in terms of the approach angles of the microscope tothe anatomy. Once specifying these approach angles, he could superposethe simulated graphics next to the microscope view. There are importantconceptual differences between Kelly's method and the method discussedhere in the present invention. First, Kelly does not use information,qualitative or quantitative, in the camera view or microscope view tomake the correspondence, registration, or quantification of what is seenin the camera view relative to the graphics data. Secondly, he neveruses two cameras to quantify the field of the cameras and relate them tothe graphic display. Thirdly, he does not use object-related or fiducialidentification points seen in one or more camera views to register theviews directly to the image data. Thus, Kelly's approach differs in afundamental way from what is being claimed in this invention.

The present invention includes in its scope the use of one or morecameras in the context illustrated by FIGS. 1 through 7. It includes theuse of a camera together with a computer and computer graphic means toregister and relate optical viewing to image data from other scanningand imaging devices. It also relates to the use of such optical andimage data correspondences to register and quantify the position ofsurgical tools such as the space probe or the microscope illustrated inthe above examples. It is related to making the associated mathematicaltransformation from a coordinate system or perspective view seen by oneor more cameras to a stereotactic coordinate system related to imagedata or a corresponding reconstructive perspective view of image dataand associated coordinate information from such image data. It relatesto the correspondence between objects both anatomical or surgical in acamera view to objects either anatomical or of an index or marker natureas represented from scanner data or extrapolated from scanner data in acomputer or computer graphic system. This was given as a specificexample from FIG. 1 in the relationship of a mechanical space pointer toindex marks and these, in turn, to corresponding quantitative positionsin space where index marks are known from image data. Registration ofthe camera viewing data to the image data may or may not involve indexmarks, index lines or index localizer devices. It may be done asillustrated in FIGS. 6 and 7 by visual or computer theoreticoptimization of registration of camera and image data or camera andreconstructed image data or enhanced camera or manipulated image data.The invention further generalizes the concept of “a camera” to othercamera-like devices. These might include an x-ray camera or an x-raysource which is point-like and projects through the anatomy to give theimage on a detection plane at the opposite side of the anatomy. Thisdata could be projectively reconstructed as though it were reflectedlight from a single camera as illustrated in the examples above. Thus,the invention subsumes the field of generalized camera viewing orprojected image acquisition relative to CT, MRI or angiographyacquisition from other imaging means and the registration thereafter tomake correspondence between these two image acquisition modalities.

Using more than one camera enables the determination ofthree-dimensional coordinates and depth of perception. The examples onFIGS. 1 through 4 illustrate this by use of a probe with two fiducialpoints on it that can be seen and digitized by the two camera views.This invention relates to the use of a video camera to quantitativelyrelate to graphic display data taken from other imaging means. Thecorrespondence of the data are illustrated by the embodiments above andthe discussion above, but those skilled in the art could think of otherimplementations of the same invention concept. For example, the use oftwo fiducial points on the probe can be extended to other types ofoptical fiducial means such as lines, other arrays of points, othergeometric patterns and figures that can be recognized easily by computergraphics, artificial intelligence, etc. The two points illustrated inthe figures could be replaced by a line of light and one or morediscrete points to encode the direction of the object. The object itselfcould be recognized by the computer graphics as a line merely by havingit of a reflective material or a particular color. The space pointer,for instance, could be white or green and thus show up differently onthe TV cameras and in the video display so as to recognize it as thepointer.

What is claimed is:
 1. A process for providing a simultaneous display of representations showing the positional relationship of a surgical instrument in a computer graphics display of a patient's anatomy, said process using stereotactic image-scanner data representing the patient's anatomy and referencing the stereotactic imagine-scanner data in scanner-data coordinates, said process comprising the steps of: storing the stereotactic image-scanner data representing the patient's anatomy and referenced in scanner-data coordinates; optically sensing plural different fields of view of the patient's anatomy in real time with a plurality of cameras each of said different fields of view containing a surgical field of the patient's anatomy and the surgical instrument, whereby to provide location data on the surgical instrument relative to the patient's anatomy and referenced in camera coordinates; transforming the location data on said surgical instrument referenced in the camera coordinates to transformed surgical-instrument data referenced in coordinates other than said camera coordinates; combining the transformed surgical-instrument data and the stereotactic image-scanner data to form combined display data referenced in said scanner-data coordinates to provided display signals; and driving a computer graphics display with the combined display data to simultaneously display an image currently representative of the surgical instrument related to the patient's anatomy.
 2. A process according to claim 1, further comprising a step of: providing index markers relative to the patient's anatomy and scanning the patient's anatomy to provide said stereotactic image-scanner data with index data in the image-scanner data.
 3. A process according to claim 2, wherein the step of transforming includes an operation of relating the camera coordinates with the scanner data coordinates by use of said index markers.
 4. A process according to claim 2, further comprising a step of: fixing a mechanical holder to the patient's anatomy and providing index markers on said mechanical holder for referencing the patient's anatomy.
 5. A process according to claim 4, further comprising a step of; referencing the mechanical holder to the location data on the surgical instrument.
 6. A process according to claim 5, further comprising a step of: repeatedly referencing the mechanical holder to the location data to correct for any positional change in the patient's anatomy.
 7. A process according to claim 6, wherein the step of optically sensing includes the step of providing at least two light sources attached to said surgical instrument.
 8. A process according to claim 1, wherein the step of optically sensing includes providing at least two light sources attached to said surgical instrument.
 9. A process according to claim 8, further comprising the steps of: providing index markers relative to said patient's anatomy prior to scanning said patient's anatomy to provide index data in said image-scanner data; and fixing a mechanical holder to said patient's anatomy, and connecting said mechanical holder to said index markers for referencing the patient's anatomy.
 10. A process according to claim 1, further comprising the step of: attaching a mechanical holder to the patient's anatomy, and connecting the mechanical holder to reference points on the patient's anatomy to provide reference data for referencing the patient's anatomy in the camera coordinates.
 11. A process according to claim 10, further comprising the steps of: referencing the mechanical holder to the location data on the surgical instrument.
 12. A process according to claim 11, further comprising the step of: repeatedly referencing the mechanical holder to the location data to correct for any positional change in the patient's anatomy.
 13. A process according to claim 12, wherein said step of referencing the mechanical holder includes: providing light sources attached to the mechanical holder.
 14. A process according to claim 12, wherein the step of referencing the mechanical holder includes; providing a pattern of light reflectors attached to the mechanical holder.
 15. A process according to claim 11, further comprising the step of: providing a pattern of light sources attached to said mechanical holder.
 16. A process according to claim 1, wherein the step of optically sensing includes providing a pattern of light sources attached to the surgical instrument.
 17. A process according to claim 1, wherein the step of: optically sensing includes providing a pattern of light reflectors attached to said surgical instrument.
 18. A process according to claim 1, wherein said surgical instrument is a microscope, and said step of optically sensing further includes the step of: providing a pattern of light sources or light reflectors attached to said microscope.
 19. A process according to claim 1, wherein the transforming step further comprises the step of: relating said coordinates by use of natural anatomical landmarks. 